fix(gorilla-merger): custom StoreAPI over tsdb.DB (bypass thanos TSDBStore crash, #39) - #317

The crash

The merger ingests fragments fine and ships 2h blocks to S3 fine, but when thanos-query issues a Series RPC the merger fatally crashes:

runtime: out of memory: cannot allocate ~8EB
  thanos/pkg/store/labelpb.ReAllocZLabelsStrings  (intern=false -> string(noAllocBytes(l.Name)))
  <- store.(*resortingServer).Send
  <- store.(*TSDBStore).Series

Root cause: under Prometheus's default stringlabels build, labels.Labels is a packed struct{ data string } (verified: sizeof(labels.Labels) == 16), not a []Label slice (24 bytes). thanos store.TSDBStore.Series:

1. builds storepb labels via labelpb.ZLabelsFromPromLabels, a zero-copy *(*[]ZLabel)(unsafe.Pointer(&lset)) reinterpret that assumes the []Label layout — on the packed-string layout it reads the 16-byte struct as a 24-byte slice header, producing ZLabels with garbage string lengths;
2. wraps the stream in a resortingServer whose Send calls ReAllocZLabelsStrings(.., false) -> string(noAllocBytes(name)), which then tries to materialize a ~8EB string from a corrupt length -> OOM.

thanos v0.41.0 is the latest release and requires Go 1.25, so there is no version/toolchain fix. Two earlier fixes are already merged (Info-service registration #313, stop double-stamping external labels), but the crash is inside thanos's own TSDBStore.Series path so it persisted.

The fix — custom StoreServer

Replace store.NewTSDBStore with a thin custom storepb.StoreServer (customStore) over the embedded *tsdb.DB. It avoids both unsafe paths:

• Series: ChunkQuerier(MinTime, MaxTime) -> Select(ctx, sorted=true, hints, matchers...). For each series, append the merger's external labels once via labelpb.ExtendSortedLabels (dedup, external wins; built with a labels.Builder so it's a normal allocation that outlives the querier), then build the storepb.Series ZLabels by copying each Name/Value (zLabelsCopy) — never ZLabelsFromPromLabels. Chunks are emitted as raw storepb.AggrChunk{Raw: &Chunk{Type: XOR, Data: <copied bytes>}} per chunks.Meta (storepb encoding = tsdb encoding - 1). Each is sent directly via srv.Send(storepb.NewSeriesResponse(...)). No resortingServer/newFlushableServer/newBatchableServer wrapper — Querier(sorted=true) already yields series ordered by label set and the consistent external-label append preserves that order, so ReAllocZLabelsStrings is never invoked. SkipChunks sends labels only.
• LabelNames / LabelValues: query the request range and merge the external label names/values (external label's only value is the merger's).
• Info: unchanged behaviour (external label set + min/max time + TsdbInfos), now sourced from customStore via the same safe copying path (ZLabelSetsFromPromLabels, which copies field-by-field).

Tests

New customstore_test.go drives the Series RPC in-process with a fake storepb.Store_SeriesServer that collects responses (no real gRPC connection needed):

• TestCustomStoreSeriesRoundTrip (the key proof): ingest two series through the real HTTP ingest handler, then call Series. Asserts (1) it does not crash/OOM and returns no error, (2) returned labels == metric labels + attrs + external labels once, no duplicates, (3) series are sorted by label set, (4) the returned XOR chunks decode back to the exact ingested samples.
• TestCustomStoreSeriesSkipChunks: SkipChunks=true returns labels (external appended once) and zero chunks.
• TestCustomStoreSeriesExternalLabelGate: a matcher on an external label with a non-matching value yields 0 series; a matching value yields 1 (matcher satisfied by the appended label, not passed to the querier).
• TestCustomStoreLabelNamesValues: LabelNames/LabelValues merge the external label and return the stored ones.

go build ./..., go vet ./..., and go test ./... -count=1 all pass (built with GOPRIVATE=github.com/ProjectASAP/* for the private asap-gorilla-go module).

Test plan

• cd gorilla-merger && GOPRIVATE='github.com/ProjectASAP/*' go build ./... && go vet ./... && go test ./... -count=1
• Deploy and confirm a live thanos-query Series RPC against the merger no longer OOMs and returns the open-window data.

🤖 Generated with Claude Code